Planning Multimodal Discourse (Invited Talks Abstract)

In this talk, we will, show how techniques for planning text and discourse can be generalized to plan the structure and content of multimodal communications , that integrate natural language, pointing, graphics, and animations. The central claim of this talk is that the generation of multimodal discourse can be considered as an incremental planning process that aims to achieve a given communicative goal. One of the surprises from our research is that it is actually possible to extend and adapt many of the fundamental concepts developed to date in computatational linguistics in such a way that they become useful for multimodal discourse as well. This means that an interesting methodologi-cal transfer from the area of natural language processing to a much broader computational model of multimodal communication is possible. In particular , semantic and pragmatic concepts like speech acts, coherence, focus, communicative act, discourse model, reference, implicature, anaphora, rhetorical relations and scope ambiguity take an extended meaning in the context of multimodal discourse. It is an important goal of this research not simply to merge the verbalization and visualiza-tion results of mode-specific generators, but to carefully coordinate them in such a way that they generate a multiplieative improvement in communication capabilities. Allowing all of the modalities to refer to and depend upon each other is a key to the richness of multimodal communication. A basic principle underlying our model is that the various constituents of a multimodal communication should be generated from a common representation of what is to be conveyed. This raises the question of how to decompose a given communicative goal into subgoals to be realized by the mode-specific generators, so that they complement each other. To address this problem, we explore computational models of the cognitive de-eision process, coping with questions such as what should go into text, what should go into graphics, and which kinds of links between the verbal and non-verbal fragments are necessary. In addition, we deal with layout as a rhetorical force, influencing the intentional and attentional state of the discourse participants. We have been engaged in work in the area of multimodal communication for several years now, starting with the HAM-ANS (Wahlster et al. 1983) and VITRA systems (Wahlster 1989), which automatically create natural language descriptions of pictures and image sequences shown on the screen. These projects resulted in a better understanding of how perception interacts with language production. Since then, we …